Skip to content

MW-1482: requisition-grain submission date, and Skipped as its own reporting outcome - #6

Merged
denys1204 merged 5 commits into
mainfrom
MW-1482-marts
Sep 11, 2026
Merged

MW-1482: requisition-grain submission date, and Skipped as its own reporting outcome#6
denys1204 merged 5 commits into
mainfrom
MW-1482-marts

Conversation

@denys1204

Copy link
Copy Markdown
Contributor

What this is

The core dbt marts for MW-1482 round 4, split from the Superset assets so the
data layer can be read on its own. The assets sit in a follow-up PR based on this
branch. Five commits, one per change.

Every requisition gets its own submission date. mart_stock_status had none,
so the round-4 dashboard work had to join one in at the dataset layer on
(facility, programme, period) - a key coarser than a requisition, so a
requisition could inherit a date belonging to a different one and an INITIATED
requisition could land in an on-time bucket. The mart now carries
submitted_date at requisition grain, from the requisition's own earliest
SUBMITTED status change.

Reporting cadence on the requisition mart. schedule_type through the shared
macro, so cadence is defined once across the marts rather than derived per chart.

The completeness column reaches the logistics mart. The newest month is
structurally incomplete, around 7% of the usual facility coverage, so it reads as
a collapse. The mart now carries in_complete_month; the chart-side filter that
uses it is in the asset PR.

Skipped becomes its own reporting outcome. Did-not-report was mixing four
things: nothing filed, filed and skipped, filed and still open, and filed under a
status we do not recognise. reporting_status is now three-valued, Reported
winning over Skipped, resolved through the SKIPPED status change for symmetry
with the existing submitted CTE. On the dev replica HIV reads 1.8% skipped
against 5.6% actually silent instead of one 7.4% block.

reporting_rate is deliberately unchanged - it counts only Reported over all
obligations, so a skip still reads as a miss there and that metric does not move.
mart_non_reporting_facilities gains was_skipped; those rows stay in the table
because its purpose is stock visibility and a skipped period is the same blind
spot, but the flag lets accountability be told apart from visibility.

A changed cohort stops reading as improvement. Month completeness compared a
month's activity against the preceding three, which catches late data but not a
month whose population changed. The weekly reporting schedule stops generating
periods in April 2026, so May carried 6,072 obligations against 9,284 the month
before while reported volume held - and the pooled reporting rate jumped 51.4% to
75.8% on the final point of every trend, on the latest complete month, which is
the point everyone reads. A month must now also carry a comparable base:
obligations for the reporting family, and the activity itself for stock and
adjustments, where the two are the same measure so the test is a no-op there by
construction. Exactly one month changes state, May 2026.

One thing that moves the moment this merges

main's Reporting Rate by Program groups by reporting_status, so it gains a
third Skipped band as soon as this branch lands, before the asset PR renames its
count or gives Skipped its own colour. Same for the Malawi twin. Nothing breaks;
it just looks half-finished until the asset PRs follow.

Three ClickHouse aliases that are not cosmetic

The analyser keeps a join key's output column qualified when the name exists on
both sides, which emits a column literally called li.requisition_id or
e.facility_id and breaks the table's ORDER BY. It cost real debugging time
three times in this work, so each of those joins aliases its key and says why in
a comment.

Verification

dbt build on this branch alone: PASS=184, ERROR=0. make verify-dbt 17/0.

The branch is deliberately checkable on its own: with only this merged, importing
main's Superset assets against the new marts leaves make verify-superset at
6/0. That proves Superset still boots and the assets still import - it does not
check chart output, so read it as "nothing crashes", not "everything renders
right".

Deploying this

mart_stock_status is materialized='incremental' with
on_schema_change='append_new_columns', so a normal run adds submitted_date as
a column and leaves already-materialised rows without a value - the timeliness
charts would then read almost every requisition as having no submission date,
with no error anywhere. It needs
dbt run --full-refresh --select mart_stock_status+. The other four marts here
are table materialisations and rebuild themselves.

Not in this PR, on purpose

Almost no EPI requisition carries a SUBMITTED status change: they go INITIATED
straight to AUTHORIZED. That is verified against the source database and its
JaVers audit trail rather than inferred, and it means EPI reads as barely
reporting at all. Treating AUTHORIZED as a filing would move EPI from 0.2% to
4.3% reported across 18 charts and lands in core for every country, so it belongs
in its own ticket. Same for whether the reporting rate should drop skipped
periods from its denominator: measured at around 0.9pp on aggregates, and the
harm it would fix is already handled by was_skipped.

mart_stock_status had no submission date of its own, so the order-timeliness
charts joined one on (facility, programme, period) and a requisition could
inherit a date belonging to a different requisition for the same key. An
INITIATED requisition showed up as submitted on time.

The mart now carries submitted_date at requisition grain, taken from the
requisition's own earliest SUBMITTED status change. 3,109 dates change on the
Malawi replica, and emergency-order timeliness moves from 84% to 43% on time,
which is the real figure.

The join key is aliased to sub_requisition_id on purpose: the ClickHouse
analyser keeps a join key's output column qualified when the name exists on
both sides, which would emit a column literally called "li.requisition_id"
and break this table's ORDER BY.
Two dimensions the dashboards had no way to offer: emergency, so regular and
emergency requisitions stop being read as one population (2,651 of 169,412
follow a different process), and schedule_type through the shared macro so
cadence is defined in one place across the marts.
The newest month in the data is structurally incomplete - about 7% of the
usual facility coverage - so it read as a collapse in the report. The mart now
carries in_complete_month from the shared completeness mart and the report
filters on it.

Both join keys are aliased (tp_program_name, tp_product_name) because the
ClickHouse analyser keeps a join key's output column qualified when the name
exists on both sides, which broke this chart once already.
Did-not-report was mixing four different things: nothing filed, filed and
skipped, filed and still open, and filed under a status we do not recognise.
A facility that legitimately skipped a period read as delinquent, and every
programme in the data has periods_skippable set, so a skip is a legitimate
outcome rather than a data error.

reporting_status is now three-valued, Reported winning over Skipped, resolved
through the SKIPPED status change for symmetry with the existing submitted CTE.
On the Malawi replica HIV reads 1.8% skipped against 5.6% actually silent
instead of one 7.4% block.

reporting_rate is deliberately unchanged: it counts only Reported over all
obligations, so a skip still reads as a miss and no existing figure moves.

mart_non_reporting_facilities gains was_skipped. Those rows stay in the table,
because its purpose is stock visibility and a skipped period is the same blind
spot, but the flag lets accountability be separated from visibility. It also
picks up an explicit alias on facility_id, without which the ClickHouse
analyser emitted a column called "e.facility_id".

mart_month_completeness reads the same column to decide which months every
reporting chart draws. It counted a skip as zero before and still does; the
comment now says it measures data arriving rather than compliance, so a future
change to the rate policy does not drag it along by accident.
Month completeness compared a month's activity against the preceding three,
which catches late data but not a month whose population changed. The weekly
reporting schedule stops generating periods in April 2026, so May carried 6,072
obligations against 9,284 the month before while reported volume held - and the
pooled reporting rate jumped 51.4% to 75.8% on the final point of every trend,
on the latest complete month, which is the point everyone reads. A composition
change was reading as improvement.

A month must now also carry a comparable base: obligations for the reporting
family, and the activity itself for stock and adjustments, where the two are the
same measure so the test is a no-op there by construction. Measured: exactly one
month changes state, May 2026, and the pooled trend then ends at 51.4% in line
with the months before it. Stock and adjustments keep 33 of 34 complete months.
@denys1204
denys1204 merged commit 9a6d842 into main Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants